{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "4b33fea1",
   "metadata": {},
   "source": [
    "### Install the dependency"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ddea83c5",
   "metadata": {
    "scrolled": true
   },
   "source": [
    "!pip install polygraphy --upgrade"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a6608846",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://tensorrt-read-only:****@urm.nvidia.com/artifactory/api/pypi/sw-tensorrt-pypi/simple\r\n",
      "Requirement already satisfied: polygraphy in /usr/local/lib/python3.8/dist-packages (0.30.3)\r\n"
     ]
    }
   ],
   "source": [
    "!python3 -m pip install polygraphy --upgrade --index-url https://tensorrt-read-only:Tensorrt@123@urm.nvidia.com/artifactory/api/pypi/sw-tensorrt-pypi/simple\n",
    "!python3 -m pip install onnx_graphsurgeon --index-url https://pypi.ngc.nvidia.com  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "534ceccf",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!pip install -r ../TensorRT/tools/Polygraphy/polygraphy/backend/onnxrt/requirements.txt\n",
    "!pip install -r ../TensorRT/tools/Polygraphy/polygraphy/backend/onnx/requirements.txt\n",
    "!pip install colored"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5415ccb5",
   "metadata": {},
   "source": [
    "## We start from an onnx model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5aed3a12",
   "metadata": {},
   "source": [
    "## 1. Inspect the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "81e44b85",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!polygraphy inspect -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7858a62e",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy inspect model resnet18.onnx --mode=basic --model-type=onnx"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd68df05",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy inspect model resnet18.onnx --mode=basic --model-type=onnx --display-as=trt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b138b60",
   "metadata": {},
   "source": [
    "## 2. Run inference using trt runtime and onnx runtime and save the results. Checking NaN and Inf."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6b74688",
   "metadata": {},
   "source": [
    "### 2.1 Using the random data as the input data.  Here, the result of onnx runtime is treated as ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d521552",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy run resnet18.onnx --onnxrt --onnx-outputs output_0 197 --save-results=testonnxrt.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0dfa16b",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy surgeon sanitize -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ef2d508d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy surgeon sanitize resnet18.onnx --fold-constants -o resnet18_new.onnx "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b5ea1e6f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy run resnet18_new.onnx --onnxrt --onnx-outputs output_0 197 --save-results=testonnxrt.pkl"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8d65ee1",
   "metadata": {},
   "source": [
    "!polygraphy run resnet18.onnx --onnxrt --onnx-outputs mark all --save-results=testonnxrt.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9dfe8226",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!polygraphy run resnet18_new.onnx --trt --validate --trt-outputs output_0 197 --save-results=testtrt.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "419bb747",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy inspect data testtrt.pkl --show-values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b7452532",
   "metadata": {},
   "outputs": [],
   "source": [
    "!polygraphy inspect data testtrt.pkl --show-values --histogram"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "449eb89f",
   "metadata": {},
   "source": [
    "### 2.2 Using the \"data_loader.py\" script to load the real input data. Here, the result of onnx runtime is treated as ground truth."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b8e09a0",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy run resnet18_new.onnx --onnxrt --onnx-outputs output_0 --data-loader-script=data_loader.py --data-loader-func-name=images --save-results=testonnxrt_data_loader.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0ca1651c",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy run resnet18_new.onnx --trt --int8 --calibration-cache mnist_calibration_bs10.cache --trt-outputs output_0 --data-loader-script=data_loader.py --data-loader-func-name=images --save-results=testtrt_data_loader.pkl"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e972219",
   "metadata": {},
   "outputs": [],
   "source": [
    "!polygraphy inspect data testtrt_data_loader.pkl --show-values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99ebf34f",
   "metadata": {},
   "outputs": [],
   "source": [
    "!polygraphy inspect data testonnxrt_data_loader.pkl --show-values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "857d06ee",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy inspect data testonnxrt_data_loader.pkl --histogram"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4ecb62d6",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy inspect data testtrt_data_loader.pkl --histogram"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae9bdcda",
   "metadata": {},
   "source": [
    "## 3. Control the comparison rules (atol & rtol)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b1544ec",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy run resnet18_new.onnx --trt --int8 --calibration-cache mnist_calibration_bs40.cache --trt-outputs output_0 \\\n",
    "--data-loader-script=data_loader.py --data-loader-func-name=images \\\n",
    "--save-results=testtrt_data_loader_bs40.pkl --load-outputs=testonnxrt_data_loader.pkl \\\n",
    "--atol=1e-1 --rtol=1e-1\n",
    "# --iterations 2  # Only for random input data\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3abb107a",
   "metadata": {},
   "outputs": [],
   "source": [
    "!polygraphy run resnet18_new.onnx --trt --fp16 --trt-outputs output_0 \\\n",
    "--data-loader-script=data_loader.py --data-loader-func-name=images \\\n",
    "--save-results=testtrt_data_loader_fp16.pkl --load-outputs=testonnxrt_data_loader.pkl \\\n",
    "--atol=1e-4 --rtol=1e-2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7fbdc12",
   "metadata": {},
   "source": [
    "## 4. Check how to keep the accuracy of low precision inference under defined rules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "af9c5696",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy debug precision resnet18_new.onnx --model-type onnx --fp16 --trt-outputs output_0 --precision fp32 -v\\\n",
    "--check polygraphy run polygraphy_debug.engine --trt \\\n",
    "--data-loader-script=data_loader.py --data-loader-func-name=images \\\n",
    "--load-outputs=testonnxrt_data_loader.pkl \\\n",
    "--atol=1e-2 --rtol=1e-2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a39deb6e",
   "metadata": {
    "scrolled": false
   },
   "source": [
    "## 5. Using template function to generate a python template script that defines or modifies a TensorRT network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d750c4f7",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!polygraphy template trt-network -h"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "16a01991",
   "metadata": {},
   "outputs": [],
   "source": [
    "!polygraphy template trt-network resnet18_new.onnx --model-type onnx --explicit-precision --trt-outputs output_0 -o template.py"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
